fix: record-ring orientation, ingest verdict pairing, CORS self-trust, and smoke alerts that clear - #85

Summary

Four fixes plus one tool. The CORS change is the original scope; the rest came from a systematic silent-failure audit run afterwards. Two of them corrupt data today.

1. EventStore record ring inverted after any delivery retry — LIVE

_records is newest-first, and that is a contract, not a convention: add_many uses appendleft, recent() reads a left-slice, and maxlen eviction drops from the right.

update_many rebuilt from _all_records(), which sorts ascending by sequence_no, leaving the deque oldest-first. recent() then served the oldest events, and the next appendleft evicted from what had become the newest end — writes deleting the data just written. One delivery retry, a single click, triggers it.

replace_all had a different bug in the same family, and it was the one cited elsewhere as the correct reference implementation. It reversed before truncating, and deque(iterable, maxlen=n) keeps the last n items, so over capacity it retained the n oldest records and discarded everything recent.

All three rebuild sites now route through _set_records, which truncates and then reverses, so they cannot drift apart again.

Why a suite that already exercised update_many twice never saw it: those tests read back through a fresh EventStore, and _load_from_disk re-sorts and re-reverses on the way in — silently repairing the state before anything was asserted about it. The other reader reset with batch_size: 1, where an inverted list is identical to a correct one. The three tests added here all read back through the same instance, and each was verified to fail against the original code.

2. Ingest verdicts matched by array position — LIVE

RegEngine does not answer in request order. Its webhook route appends replay-window and KDE rejections first, then rules-enforcement rejections, then every acceptance, so the response events list is partitioned rejected-first.

Zipping it against the request by index misattributes as soon as a rejection follows an acceptance: a valid lot is stored as failed carrying another lot's validation errors, and the event_id, sha256_hash and chain_hash of a genuinely accepted event are copied onto the wrong record. Those hashes are the evidence.

(traceability_lot_code, cte_type) is already on the wire on both sides and is the join key. It can legitimately repeat inside one batch — the same lot and CTE at two timestamps — so responses are held in a per-key queue and consumed in order rather than collapsed into a dict. A key with no response left reads as None, meaning no verdict. Positional fallback is deliberately gone: there was always some answer before, which is exactly why nothing ever looked wrong.

Four call sites had this, not two. Beyond step() and the CSV import, the demo-fixture load and the delivery-retry path zipped by index too — and the retry path is the worst, since a retry that succeeds for one record could mark a different record posted.

Why the existing suite is blind to it: every other test delivers through app/mock_service.py, which appends accepted and rejected in a single pass, so the mock's response is in request order. The new tests supply a genuinely partitioned response; the end-to-end one was verified to fail against the original code, storing the rejected lot as posted.

3. CORS: trust the service's own platform origin

Makes the service trust https://$RAILWAY_PUBLIC_DOMAIN in addition to whatever REGENGINE_CORS_ORIGINS resolves to. This is the code half of the August cutover lesson; the docs half landed in #84.

A union, not a fallback. In the incident the variable was set — to the previous service's URL, via a Railway reference variable — so a fallback that only applies when the variable is missing would have changed nothing. The same list also gates state-changing requests in auth_middleware, so browser writes failed too, and both nightly smokes stayed red for three days (#80, #81).

Safe: the appended origin is only the service's own canonical domain, injected by Railway. Anyone who can forge RAILWAY_PUBLIC_DOMAIN already controls the deployment. Wildcards remain rejected, and explicit origins for any other host still require configuration. A malformed platform value degrades to "no extra origin" rather than raising — this runs while the ASGI app is being constructed, and a platform-injected string must never crash startup.

4. Smoke alerts that clear

The alerts only ever fired. Nothing closed them, and the issue body said so out loud — "Close it once the smoke passes again" — making a human the entire recovery mechanism. #80 and #81 both stayed open for days after the underlying failure was fixed.

The second-order effect is the more damaging half: the tracker lookup is scoped to state: 'open', so a stale tracker from a resolved incident silently absorbs the next genuine outage as one more comment on an aged thread. The strongest signal GitHub offers — a brand-new issue — was spent exactly once, on the first failure ever.

smoke-failure-issue.yml gains a resolved input and a close path; both smoke workflows now call it on success as well as failure. It closes every match rather than the first, since duplicates predating this edge would keep the problem alive, and it skips pull requests the same way the open path now does.

Verified by running both scripts against a mock GitHub API: first failure opens an issue, a second comments without duplicating, recovery comments and closes, and — the point of the change — the next outage after a recovery opens a new issue rather than commenting on the closed one. Recovery with nothing open is a no-op, and a PR carrying the marker in its body is never closed as a tracker.

5. scripts/cutover_preflight.sh

Mechanises what the cutover checklist can verify from outside: every read-only path in the dashboard's proxy contract answering alike on both services, the new service reporting GitHub-injected build identity rather than a stale REGENGINE_BUILD_SHA, and the new service trusting its own origin. Read-only by construction — the demo is shared, and the POST routes the dashboard proxies mutate its state.

The more important half is what it refuses to claim. The Basic-auth credentials live as secrets on two platforms, so from outside a service with the wrong credentials is indistinguishable from one with the right ones — both answer 401. Rather than let a green run imply a clean bill of health, the success path names that gap and directs the post-flip check at /api/simulate/status rather than /api/healthz, since the proxy answers 200 with {"offline":true} when the backend is unreachable.

Test Plan

• Full suite: 150 passed locally
• All three record-ring tests verified to fail against the original code; they read back through the same instance, which is the blind spot that let this ship
• Verdict-pairing end-to-end test verified to fail against the original code (stored the rejected lot as posted)
• Both issue alerters run against a mock GitHub API across six scenarios, including post-recovery re-alerting and PR-body false positives
• Pre-flight run against the live service pair: passes on the real cutover target, and both failure paths exercised (non-GitHub service trips build identity; unreachable host trips contract and CORS) with non-zero exit
• pytest and browser-smoke green on 0c428a5